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We study theoretically the electron correlation effect in a three-dimensional (3D) Dirac fermion system which de- 
scribes a topologically nontrivial state. It is shown within the mean-field approximation that time-reversal and inversion 
symmetries of the system are spontaneously broken in the region where both spin-orbit coupling and electron correlation 
are strong. This phase is considered as an analog of that in the lattice quantum chromodynamics (QCD). It is also shown 
that in the presence of magnetic impurities, electron correlation enhances the appearance of the Weyl semimetal phase 
between the topological insulator phase and the normal insulator phase. 

KEYWORDS: topological insulator, Weyl semimetal, spontaneous symmetry breaking, electron correlation, Wil- 
son fermion 



Recent discovery of topologically nontrivial phases has 
moved modern physics to a new direction. 1 - 2 Especially, the 
experimental realization of 3D topological insulators 3 ' 4 has 
accelerated the researches in this field. It is known that strong 
spin-orbit coupling is essential to realize a topologically non- 
trivial state. In the presence of inversion symmetry, this non- 
trivial state is characterized by the change in the parity of 
the lowest unoccupied band from even to odd due to strong 
spin-orbit coupling. On the other hand, strong electron corre- 
lation has been a central issue in condensed matter physics. 
Recent years, intensive studies have been done in systems 
where both strong spin-orbit coupling and strong electron cor- 
relation exist, triggered by the discovery of a novel Mott- 
insulating state in iridates. 5-8 The motivation of these studies 
are in the searches for novel phases due to the interplay be- 
tween spin-orbit coupling and electron correlation. Preceding 
studies of the electron correlation effect on topological insula- 
tors have mainly focused on the competition between the spin 
(or charge) ordered phase and the topological insulator phase 
of Hubbard-like models on honeycomb lattices, 916 other 2D 
lattices 17-19 and 3D lattices. 20-23 

Another topological phase, the Weyl semimetal, where 
gapless linear dispersions exist in a 3D system, has gathered 
attentions these days. Near the band-touching points (Weyl 
points), the excitations are described by the massless Dirac- 
Weyl Hamiltonian. This quasiparticle, the Weyl fermion, is 
assigned a chirality, and the bulk band gap opens only if the 
two Weyl fermions with opposite chirality meet each other. 
This topological behavior originates in the nonzero Berry cur- 
vature enclosing a Weyl point. 24 The realization of the Weyl 
semimetal phase requires breaking of either time-reversal 
or inversion symmetry. 25-29 It is remarkable that the Weyl 
semimetal phase is predicted in strongly correlated pyrochlore 
iridates. 24 This suggests that topological phases emerge in 
strongly correlated if -electron systems. 

In this Letter, we focus on the electron correlation effect in 
a 3D Dirac fermion system which describes a topologically 
nontrivial state, and analyze it within the mean-field approx- 
imation. We assume the strength of the Coulomb interaction 
between the bulk electrons as a constant for simplicity, al- 



though it would be correct to adopt 1 jr Coulomb interaction 
because the screening effect is considered to be weak in Dirac 
fermion systems. Further we consider the case that the mag- 
netic impurities are doped into the system, and discuss the 
emergence of the Weyl semimetal phase. 

Let us consider a 3D model on a cubic lattice. We start from 
the 3D Wilson fermion, which Hamiltonian is given by 



^i^(fc) • a M 

M =i 



with 



Rj(k) = sin kj, 

/?4(fc) — mo + (l - cos kjj , 
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crj 


, a 4 = 





-1 


, a 5 = 
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where j (= 1,2, 3) denotes spacial axis, r > 0, and a v are the 
standard (Dirac) gamma matrices, 



(3) 



Here crj are the Pauli matrices. These matrices satisfy the anti- 
commutation relation {a^, a v ) = 25^. The energy of this sys- 
tem is measured in unit of vp/a with vf and a being the Fermi 
velocity and the lattice constant, respectively. The Wilson 
fermion was originally proposed as a model which describes 
lattice fermions without doublers. Although 2 3 = 8 fermion 
doublers are generated due to the priodicity of sin kj, the Wil- 
son term r 2 ,-(1 - cos kj) eliminates them. Thus when mo = 0, 
massless Dirac fermion appears only at the point k = (0, 0, 0). 
On the other hand, the Hamiltonian (1) with mo < can be 
regarded as a simplified one which describes 3D topological 
insulators such as Bi2Se3. 4 In Eq. (1), the spinor is written in 

the basis of [ c feXT' c Ia1' c feBt' c 1bJ' wnere ct ' s ^e creation 
operator of an electron, A, B denote two orbitals, and f (X) 
denotes up- (down-) spin. 4 

In the presence of time-reversal symmetry and inversion 
symmetry, the Zi invariant of the system is given by 30 ' 



(■ 



-i) v =n< _sgn ^ 4(Ai)] '' 



(4) 
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(a) 



where A,- are eight time-reversal invariant momenta. It is eas- 
ily shown that if > mo > — 2r or — 4r > mo > — 6r (;7?o > 0, 
— 2r > mo > — 4r, or — 6r > mo), the system is topologically 
nontrivial (trivial). 

Next we consider the Coulomb interaction between the 
bulk electrons, which is given by 

Hnt = V ^ C l + qa C l-qfi C k'fi C ka, (5) 

k,k f ,q afi 

where a and B denote the component of spinor, (A, f), (A, X), 
(B, T), (B, |). Although V(q) = An/q 2 in the original form, we 
assume that V(q) — V — const, for simplicity. In the mean- 
field approximation, only the Fock term gives the correction 
to the mass term as follows: 

^ F = 2N X [( c L c w) ( c U Ck ' a ) - 2 ( c L c w) c U c i«'«] ■ 
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(6) 

where we have retained only the terms which satisfy q = k' - 
k. Here we assume that A a/3 = jjXk ( C L C ^) = ( A 4 a 4 + 
&5&5)a/3- This assumption is based on that in QCD, uj/ a ij/pS = 
(crl + i7rys) a fi with cr and n being the chiral condensate and 
the pion condensate, respectively. Then Eq. (6) becomes 

?C F = 2NV (Aj + A 2 ) - Vj] c{ (A 4 a 4 + A 5 a 5 ) c k , (7) 

k 

where we have defined c[ = [cj^, cj^, c[ BV cj^J. 

Combining Eqs. (1) and (7), we obtain the effective Hamil- 
tonian 



TfeffCfc) 



5 

J]R M (k)-a M , 



(8) 



where Rj(k) = sinfcy, R4(k) — mo - VA4 + j (l _ cosfc,), 
and Rs(k) = -VA5, The energy eigenvalues of the effective 

Hamiltonian (8) is given by ±E k = + -Jl^=i [^(^)] ■ Let us 
express the eigenfunctions of Eq. (8) as where + denotes 
the positive or negative energy eigenvalue and A (= 1,2) de- 
notes the degeneracy of the energy eigenvalue. Then c k and 



c' k can be written as 



c k 



,1 

YHA u U +b L( u U]> 



(9) 



where ( (frj^J is the creation operator of an electron in the 
positive (negative) energy band. Then the mean-field Hamil- 
tonian of the system is given by 



H = 2NV(A 2 + A*) + Yj cl%«(k)c k 

k 

= 2NV (A 2 + A 2 ) + J] [£fcflL fl fe-i - ^lA*] • 



(10) 



In this paper, we consider the case of zero temperature and 
set the Fermi energy e F = 0. In this case, (aj^afc',1') = and 




k x /ir 



k x /-7T 




Fig. 1. (Color online) (a) Phase diagram. The phase boundary between the 
topological insulator phase and the normal insulator phase is determined by 
the condition m c (f = or m e (j = — 2r. The phase boundary between the A5 # 
phase and the other phase is determined by the condition A5 = 0. The in- 
dependence of surface spectra with k y = at (mo, V) = (-2.5, 3.7), (-2.5, 4.3) 
and (-2.5, 5.5) are shown in (b), (c) and (d), respectively. 



(b' kl bk'A^j = #fe,fc'£/W are satified. Thus it follows that 



k.A 



= - y — [E k - t H(k)] . 
N L—i 2Ek 

k 



(ii) 



The values of A4 and Ag are obtained by solving the following 
self-consistent equations: 

R 4 (k) 



A 4 = itr(a 4 A) = -i^ 



N k 2Ek 



(12) 



2E k 



Solving these two equations is equivalent to obtaining the sta- 
tionary point of the free energy per unit volume of the system 
F(A4,A 5 ): 

aF(A 4 ,A 5 ) dF{A A ,A 5 ) 



dA 4 



dA 5 



0, 



where 



F(A4, As) = 2V (A 2 4 + A 2 ) - i E * 



(13) 



(14) 



In the effective Hamiltonian, we define the effective mass: 

m eff = m a - VA 4 . (15) 

The phase diagram is shown in Fig. 1(a). Throughout this 
paper, we set r = 1. Although this value is far from that 
which was obtained by an ab initio calculation 4 for Bi2Se3, 
we have confirmed that the value of r does not change the 
results largely. It was found that A4 is always negative, and 
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the effective mass is a monotonically increasing function with 
respect to the electron correlation strength V. As a result, 
there exists the critical strength V c where the effective mass 
changes its sign in the mo < region. As mentioned above, 
if > OT e ff > -2 (m e ff > or -2 > m e s > -4), the system 
is identified as a topological insulator (normal insulator). The 
value of mo for Bi2Se3 is estimated as about -0.3. 

We see that two values of A5, namely A5 = or A5 = +c 
(c is real), can exist as the solutions of Eq. (12). The phase 
where A 5 + is realized if F(A 4 , A 5 + 0) < F(A 4 , A 5 = 0). 
For example, on the mo = -2.5 line, A5 starts from to 
take nonzero value with increasing V, and reaches the max- 
inum absolute value +0.098. Then A5 decreases toward zero. 
The surface spectra in the topological insulator phase and the 
A5 + phase are shown in Fig. 1(b) and Fig. 1(c), (d), re- 
spectively. In the A5 + phase, the energy gap of surface 
modes opens due to nonzero A5 and thus the system is insu- 
lating. From the symmetry point of view, the matrix as breaks 
both time-reversal (T) symmetry and inversion (J) symmetry, 
i.e., Ta^l 1 = -0:5 and Ia 5 I~ l = -as are satisfied, where 
7~ = 1 ®{-i<j2yK and I = 0-3 ® 1. This means that the sponta- 
neous symmetry breaking occurs due to electron correlation. 
It is known that a similar phase (in which parity and flavor 
symmetry are spotaneously broken) exists in the lattice QCD 
with Wilson fermion. 32-34 In Fig. 1(a), the A5 + phase is re- 
alized in the region where both \mo\ and V are large. Negative 
sign of mo originates in the level crossing of the two orbitals 
induced by spin-orbit coupling. Therefore, the value of \mo\ 
can be considered as the strength of spin-orbit coupling. 4 It 
can be said that this novel phase may emerge in systems where 
both spin-orbit couling and electron correlation are strong. 

Next we consider to add a time-reversal symmetry breaking 
term. 26 This term can be regarded as an external magnetic 
field along the z-axis or magnetic impurities. In this case, the 
effective Hamiltonian is written as 




Fig. 2. (Color online) Phase diagram calculated with the Z12 term, (a) In 
the case of mo = -0.30, (b) mo = +0.30. The topological insulator (TI) phase 
and the normal insulator (NI) phase are determined by the sign of the effective 
mass. The Weyl semimetal (polarized insulator (PI)) phase is obtained as the 
phase in which the Weyl points exist (do not exsist). The k. dependence of 
the energy bands (Eq. (17)) in the case of mo = -0.30 with (k x ,k y )=(0,O) at 
(b. V) = (0.10, 0.30), (1.0, 0.30) and (1.90, 0.30) are shown in (c), (d) and (e), 
respectively. 



(16) 



where £12 = — i [ai, aj\ = 1 ® (X3. We can obtain the eigen- 
values of this effective Hamiltonian analytically, which leads 
to 



E(k) 



\ 



Z K(fc)] 2 



\ 



±b 



(17) 



The Weyl points, at which the two bands touch, appear 
where the wave vectoj k satisfies the condition b 2 = 
|m eff + rYjj (l - cosfc,)] + (VA 5 ) 2 + sin 2 fc ; and sin£ v = 
sinfcy = 0. k x and k y can take the value or n. Setting \mo\ 
small, we concentrate on the Weyl points which appear on the 
(k x ,k y ) = (0,0) line. The values of A4 and A5 are obtained 
by solving two equations self-consistently: dF(A4, A$)/dA4 = 
dF{Ai[, As)ldA^ = 0. Here F is the free energy per unit vol- 
ume at zero temperature, which is defined by 



1 2 

fXA4,A 5 ) = 2v(A 2 + A 2 ) + -£^4, 



(18) 



k i=l 



where are the engative two energy bands of Eq. (17). 
The phase diagram calculated with the £12 term is shown 



in Fig. 2. First we set mo = -0.30, intending Bi2Se3. [see 
Fig. 2(a)]. As in the case of b = 0, the effective mass m e g 
is a monotonically increasing function of V. We see that the 
Weyl semimetal phase emerges between the topological and 
normal insulator phases, as the perturbation term b increases. 
Although the Hamiltonian (16) does not have time-reversal 
symmetry, we call the region where the effective mass is neg- 
ative the topological insulator phase. The transition from the 
topological insulator phase to the other phase occurs when the 
band gap closes. Thus as far as the band gap is open, we call 
the phase in which the effective mass is negative the topo- 
logical insulator phase, regardless of the loss of time-reversal 
symmetry. 

The Weyl semimetal phase is defined as a phase where the 
gapless dispersions exist on the fc z -axis. In this phase, the ef- 
fective mass changes its sign. However, once the band gap 
closes, the sign of the effective mass has no importance. At 
fixed values of b, the Weyl points first arise at k z = with 
increasing V. The Weyl points split and start to move toward 
a certain value of k z , then go back to k z = and the band 
gap opens. At fixed values of V, the Weyl points first appear 
at k z = with increasing b, then the Weyl points split and 
start to move toward k z = +n. These points disappear af- 
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Fig. 3. (Color online) Phase diagram with b = 0.20. In the presence of 
magnetic impurities, the regions where the absolute value of m c ff and m e ff +2r 
are small in Fig. 1 (a) changes to the Weyl semimetal phase. 



ter reaching k z - ±n and the band gap opens. We call this 
state the polarized insulator, because this phase is realized 
with large b. When V = 0, we can obtain the value of the 
phase boundary analytically: b = \mo\ = 0.30 {k z = 0) and 
b = mo + 2r = 1.70 (k z = n). The four energy bands (Eq. 
(17)) at (b,V) = (0.10,6.30), (1.0,0.30) and (1.90,0.30) on 
the phase diagram (Fig. 2(a)) are plotted as a function of k z in 
Fig. 2(c), (d) and (e), respectively. 

Next we set mo = +0.30 [see Fig. 2(b)]. In this case, 
the effective mass m en " always takes positive value and m e ff 
is a monotonically increasing function of V. Thus the Weyl 
semimetal phase begins to arise with finite b due to the con- 
dition b 2 = |m e ff + rj^j (l - cosfcy)] + sin 2 k z , in contrast to 
the case of mo = -0.30. The behavior of the Weyl points with 
respect to b and V is the same as the case of mo = -0.30. 
When V = 0, we can obtain the value of the phase boundary 
analytically: b = mo = 0.30 (k 7 = 0) and b = mo + 2r - 2.30 
(k z = n). 

Finally, the phase diagram with b fixed to 0.20 is shown 
in Fig. 3. When b is small, there exists the Weyl semimetal 
phase only in the mo < region. This emergence of the Weyl 
semimetal phase results from the change in the sign of m e ff 
and m e s + 2r. We see that the A5 + phase is robust against 
the perturbation b. If A5 is not zero, the band gap opens and 
therefore the Weyl semimetal phase cannot exist. 

In a realistic system, for example in Bi2Se3, the value of 
mo is estimated as about -0.3. Our calculation shows that the 
critical strength V c at which the Weyl semimetal phase ap- 
pears is about O.ltivp/a = 0.7 [eV], when = 3 [eV-A] 
and a = 3 [A] are assumed. This value is considered to be 
too large for /^-electron systems. Therefore, a possible way to 
realize the Weyl semimetal phase in 3D topological insulators 
such as Bi2Se3 is to make the bulk band gap (^ |mo|) smaller 
without the big change in the strength of Coulomb interaction 
between the bulk electrons. This experimental procedure has 
been realized in a solid-solution system BiTl(Si_ A Se A )2. 35 ' 36 

To summarize, we have studied the electron correlation ef- 
fect in a 3D topological insulator which is described by the 
Wilson fermion within the mean-field approximation. As the 
correlation strength increases, the topological insulator phase 
changes to the normal insulator phase and vice versa. In the 
region where both spin-orbit coupling and electron correla- 



tion are strong, time-reversal and inversion symmetries of the 
system are spontaneously broken. This phase is considered as 
an analog of that in the lattice QCD. By adding a time-reversal 
symmetry breaking term, the Weyl semimetal phase emerges 
between the topological and normal insulator phases. In our 
calculation, the bulk band gap monotonically increases and 
becomes infinity in the limit of strong correlation. The anal- 
ysis of not only the bulk but also the surface states from the 
strong correlation limit is needed to understand the correation 
effect in 3D topological insulators. 
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